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ABSTBACT 

This paper discusses the probleas of gathering and 
analyzing data on classrooa behaviors. The difficulties involved in 
using the cojaaon pretest^posttest experimental design are described, 
lo grapple vith these problems, a study was conducted on t^e effect 
of environmental changes on student behavior^ employing 
design and time-series analysis. The hypothesis tested vas that minor 
changes in the physical setting of the classroom could produce 
predictable, desirable changes in students' behavior* Formal 
observations of students' locations and activities vere conducted six 
times a day for tvo weeks, using a time*sampling*by*c}iild instrument. 
The data from the six daily observations of all children vere reduced 
to daily summary statistics indicating the percentage of time 
children vere observed in a particular area or engaged in a specific 
activity. The data vere then analyzed using time*series techniques. 
Besults indicated that most of .the desired behavior changes vere 
produced, thus supporting the hyppthesis. The same experiment vas 
conducted using the traditional statistical methods. The conclusion 
vas reached that time-series techniques appear to have great 
potential for educational researcii. (JD) 
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One of the most commonly tised research paradigms in benavloral science 
Is the pre-test~post-te8t experimental design Involving Independent sub- 
jects divided into experimental and control groups • The data collected In 
a study like this are typically analyzed with analysis of variance, t-test 
or chl square techniques. Such analyses are stralg^tfoKvard and valid if 
all the assumptions of the chosen technique have been met. Of particular 
importance is the requirement that the observations be conducted on inde- 
pendent subjects. 

This requirement poses a knotty problem for the educatloxial researcher 
who is observing the behavior patterns of interacting students. In this 
case an obviouiB dependency exists among subjects. In addition, the diffi- 
culty of perfectly matching classrooms makes the selection of a control 
group particularly troublesome. Since a pre-test — post-test design without 
a control group is hardly adequate, observations of the students can be 
conducted at several time periods before and after treatment. This would 
transform the pre-test— post-test situation Into a quasi-experimental 
time-series design, which would be better able to distinguish changes due 
to the treatment from those caused by a steady trend (brought about by 
history or maturation, for example) or by experimentally unplanned classroom 
events. 

Successive observations of the same subjects are clearly not inde- 
pendent, however, and do not fall into any class of ANOVA techniques which 
could normally be used to assess a treatment effect. Indeed, Scheffe (1959), 
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discussing the effects of departures from the underlying assumptions of 
ANOVA on error rates and estimation has stated: "In general, of the three 
kinds of possible departures from assumptions we have considered [normality 
of errors and effects, equality of variance of the errors, and statistical 
Independence of the errors], those caused by lack of Independence are the 
most formidable to cope with" (p. 364) • 

Research on the relationship between environment and behavior in 
general~and classroom settings and student behavior in particular — is often 
plagued by problems of dependency among subjects and over time. Perhaps for 
this reason, several studies of man-environment relations have been purely 

descriptive, with no statistical, analysis: Brunettl (1972) ; Hahn (1973) ; 

Rlviin and Wolfe (1972); and Zlfferblatt (1972). There have also been a 
nusiber of studies which ignore the independence requirement and use 
inappropriate or questionable statistical methods, such as chl square, 
t*-tests, or analysis of variance: Brookes and Kaplan (1972); Coates and 
Sanoff (1972); McGrew (1970); and Winett, Battersby,^'and Edwards 
(1975). 

In an attempt to grapple with these problems, a study by Weinsteln 
(1975), on the effect of environmental changes on student behavior, 
employed a tlme*-series design and time-series analysis.. The experimenter 
observed the spatial distribution of activity in a second-third grade 
informal classroom divided into various interest areas, before and after a 
change in room arrangement. She tested the general hypothesis that minor 
changes in the physical setting could produce predictable, desirable 
changes in students* behavior. Formal observations of students* locations 
and activities were conducted six times a d ^ for two weeks using a 
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tlme-sanrpllng-by-chUd Instrument* Data were recorded on a floor plan of 
the room. A descriptive analysis of this pre-change data revealed several 
problems with the way the room and the materials In It were being used 
(e.g. certain areas were crowded, while others were under-used; students 
were not Involved with the manipulative materials to the extent the teacher 
desired). Design changes were then made with specific behavioral goals in 
mind, and the two-week post-change observation period was begun. 

2 

The widely used statistical techniques— ANOVA, t-test and X were 
found to be inadequate to assess the effect of the design changes. Serial 
dependency among observations for one subject negated the use of the t-test, 
_ ^SLVid the variation in correlation among the series of observations negated 
use of repeated measure ANOVA. The chi square test could be used if it was 
assumed that entries in an area population-by-pre-post contingency table 
were. a sanple of time periods (thus achieving independence of the observations, 
which could not be assumed if the observations were a sample of subjects). 
Still, the chi square test could not distinguish between a change due to 
the intervention and a change due to a steady trend. 

With these difficulties in mind, tim^-series analysis (Box and Jenkins, 
1970) appeared to be the most satisfactory solution. The data from the 
six daily observations of all children were reduced to daily summary 
statistics indicating the percentage of time children were observed in a 
particular area or engaged in a specific acti\^ty. The resultant series 
of pre- and post-change data points, one for each day, were then analyzed 
using time-series techniques. Results Indicated that most of the desired 
behavior changes were produced, definitely supporting: the general hypo- 
thesis that relatively simple changes in room arf'angament can prodtice 
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specific, desired changes In behavior. 

Since other investigators dealing with slioilar qiiestlons had used 
statistical teclmiques rejected by the experiioanter^ the present authors 
were interested in seeing what kinds of results would have been obtained 
had standard t-tests been used~ltt" particular, the frequency of type I 
errors and estimates of the magnitude of the intervention effect. Three 
specific analytical techniques were considered. The first was the standard. 
Independent, two-sample t-test assuming equal population variances, and 
estimating a shift by the difference of post-test and pre-test series nsans. 
The second method was the basic time series analysis techniqtie promulgated 
by Box and Jenkins (1970), based on maximum likelihood estimation. The 
third method was a Bayesian time series analysis put forth by Box and 
Tlao (1965). To assess the difference among these analytical techniqCir^s , 
a series of data sets, based on the Welnstein situation, were generated. 
Each set consisted of 40 pre-test and 40 post-test observations. The 
basic model used in the Welnstein study and for the generation of these 
data sets was the integrated first order moving average model, which Box 
and Tlao (1965) recommend as a frequently encountered form of time-series 
models: 

Zl » L+ai 

«» L+y Z aj+at t»2,3,.^.,m 

(1) 

" 6+L+Y Z a-i+a^ t«in4-l,m4-2, . . . ,n 

0< Y <2 
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where {a^^} Is a series of Independent random normal deviates with mean 0 
and variance O'^^ L ±s the base levels 6 Is the change In level after time 
m — caused by the Intervention— and y ^-s proportion of past deviations 
Incorporated Into the present observation^ also called the moving average 
parameter. 

The model can be written In two other equivalent forms. Equation 2 
Indicates that the present observation Is based on the previous observation^ 
a portion of the previous deviation and a present deviation. Equation 3 
indicates that the first differences form a series based on a present 
deviation plus a portion of the prevlotis deviation. 

Zj^ « L+a^ 

(2) 

2|. * Zj..! + (Y-^l)at-i+a^ t»2,3,...,m,m+2,...,h 

0< Y <2 

and 

(Zj.-Zj._l) * ^ ^t-l'^^*^t"^t-l^ t«2,3, . . . ,m,mf2, . . . ,n 

(3) 

<2mfl-V = <S + Y V^^mfl-V 

0< Y <2 
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In all, 90 sets of data were generated^ and each set was analyzed by 
each of the three techniques. The standard deviation of the took on 
values of .1, .5 and .7; 6 took values of 0, »5, 1.0, 1.5 and 2.0; and L 
was 10 for all cases. The values of Y determine the dependency an»ng the 
terms in the series. When Y is 1 there is no serial correlation among the 
first differences of the observations; that is, the present observation is 
the previous observation plus the random component a^ (see equation (2)). 
When Y is zero the present observation is a uniformly weighted sxim of 
previous observations which is an extreme correlational situation. In the 
simulations the values of Y were 0, .25, .50, ,75 and 1,0. Values in the 
range 1.0 to 2.0 were not considered as they wotild represent a negative 
weighting of past observations which is not consistent with many behavioral 
models. One set of data was generated for each combination of C7, S and Y 
yielding 75 data sets, and a second replication was done with Y = 1 for all 
combinations of O and 6 yielding an additional 15 data sets. 

What happens is a brief outline of the time series techniques, beginning 
with the method of Box and Tiao (1965) . The technique is to calculate the 
lagged correlations for the pre-test and post-test data. On the basis of 
these correlations a suitable taodel is chosen (if differencing of the 
observations is indicated, in order to remove nonstationarity, the lagged 
correlations are calculated from the differenced data) . The model is com- 
posed of some combination of autoregresslve, moving average and differencing 
terms. To analyze the simulated data, the form of the model was assumed 
to be known as a first order moving average on the first differences of 
the observations. If the parameter Y is unknown, it is not possible to 
directly estimate the value of 6. The approach is to employ a Bayesian 
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analjrais uaitig sample Information about y to make estimates of S* 

In the model (Eq, 1), there are four unknown parameters— 6 , L, 
and a. The distribution of 6 conditional on y and the observations can 
be calculated and plotted as a function of y as y ranges from 0 to 2, A 
similar conditional distribution can be calculated for the parameter L. 
Of major importance is the posterior distribution of ^ assuming no prior 
information (a uniform prior distribution for y) and the value of the t 
statistic to test the hypothesis 6«0, as y varies from 0 to 2. For 
Illustrative purposes, a plot of these functioris is shown in Figure 1 for 
the data generated when 6 =0.5, a = .5, and y = .25. The Bayesian 
analysis based on the functions in Figure 1 indicates that, if there is no 
prior knowledge of y, the posterior distribution, h(y/Z), has its support 
over the range of 6 from .17 to .75. Over this entire range the t value 
is above the 1% critical value of 2.33, indicating a 6 significantly different 
from zero. The estimated value for 5 would be in the range of 1.62 to 2.01. 
The analysis clearly indicates a non zero value for '6 and a y in the range 
of .17 to .75. 

Maximum likelihood analysis, on the other hand, would choose the value 

of Y which has the maximum probability of the posterior distribution of y. ^ 

^ .IP 

In this case It is at f =» ^S/H At this point t « 3.76 and 6 « 1.88, From 
the other functions, not graphed, it is found that the maximum likeliKood 
estimates of L and a are 10.15 and .42 respectively. The estimation is 
accurate. The t-test estimated 6 at .29 with a t value of 1.28, which 
lends to the false concltision of no change in level. 
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There is little difference in the final result (relative to the 

I 

estimation of 6 with a uniform prior) between the two time series 
technqiues. In the remaining discussion, the acceptance or rejection of 
the hypothesis that 6=0 will apply to both the Bayesian and maximum 
likelihood analysis, and the estimate of 6 and t value will be the maximum 
likelihood values. 

The results of the analyses of the 90 simulations clearly demonstrated 
the advantages of using time series techniques, for analyzing this type of 
data. In one of the six sets of simulated data with no serial correlation 
and 6 « 0, the t-test and time series tests both indicated a 6 statistically 
significantly different from zero. The t value was 2.15 (1% level), but 
the estimate of 6 was small (6 « -.06). This can be attributed to an 
expected Type I error occurrence. 

In the twelve simulations with 6=0 and serial correlation (y ?fc 1), 
the t-test analysis indicated a 6 significantly different from zero in 11 
cases; the mean of the absolute value of the t statistic was 5.44; the 
minimum value was 2.15. This corresponds to a Type I error rate of 92% 
when the testing was carried out at the nominal 5% level. The estimated 
magnitude of the shift was 1.39 with individual estimates ranging from 
4.56 to -1.25. The time series techniques indicated two of the twelve 
simulations had statistically significant estimates of 6. I^hen tests are 
carried out at the 5% level of significance, it would be expected that in 
12% of the cases at least tt^o significance values would be observed in 12 
trials. Thus, two is not an extreme number of false rejections. It should 
be noted that both of these errors occurred at the maximum O value and at Y 
values of 0 and .25. Since these represent the two most extreme trials, it 
appears that with highly variable or strongly correlated data the error rates 
may not be at their stated significance levels in this type of time series 
analysis . Q 



For the 16 cases with no serial dependence and 6 ?t 0, the t-test and 
time series teclmiques yielded results similar to each other. Both indi- 
cated the presence of the shift and estimated its magnitude to a desirable 
degree of accuracy (the magnitude of the error increasing with a). The 
remaining 48 cases of serial dependence and 6 0 indicated that the t-test - 
was seriously in error. The problem was not so much in the Type II error 
rate, since there were only four non-rejections in 48 cases when the 
alternative hypothesis v/as t\^o-sided, and 14 non-rejections for the 
appropriate one-sided hypothesis. However, the derived estimated value of 
6 is poor. For the 48 cases, the mean error of the estimate of 6 was 21.8 

A 

Standard deviation units, where the s.d. unit is | 6'"6 1 • The corresponding 

a//n 

error in 6 for the maximum likelihood time series estimate is 6.9. The 
Bayesian time series analysis failed to reject the appropriate one-sided 
null hypothesis in only 9 of the 48 cases. 

In summary, when serial correlation is present, both the t-test and 
time series techniques applied to this data have approximately the same 
Type II error rates (29% and 19%), but the estimate of the shift produced 
by the t-test is less accurate. The t-test has a Type I error rate near 
100%, when tests are performed at the nominal 5% level, while the time-series 
technique has a Type I error rate that is not statistically significantly 
different from 5%. When no serial correlation is present all techniques 
perform equally. 

It is^ important to point out that this analysis was carried out knowing 
the form of the model exactly, and only estimating the parameters. In 
actual situations, the foxna of the model must also be estimated from the 
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sample correlation coefficients « The additional estimations of a model 
could change the above stated results. 

Time series techniques appear to have great potential for educational 
research. Experimenters attempting to fulfill the assumptions of the 
time series model will Invariably produce more rigorously designed studies 
(e*g. by ensuring that they have a sufficient number of data points, that 
the Intervals between observations are equal , etc.)* In addition, aware- 
ness of these statistical procedures and their advantages xdlll hopefully 
eliminate the use of Inappropriate statistical methods and will stimulate 
research into areas which have previously been Ignored due to the lack 
of appropriate statistical tools. 
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